Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RFC: NodeOverlay #1305

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

RFC: NodeOverlay #1305

wants to merge 1 commit into from

Conversation

ellistarn
Copy link
Contributor

@ellistarn ellistarn commented Jun 6, 2024 •
edited
Loading

Fixes:

Description

---
# Reduce on-demand prices to 90%
# https://github.com/aws/karpenter-provider-aws/issues/3860
# https://github.com/aws/karpenter-provider-aws/pull/4697
kind: NodeOverlay
metadata:
  name: discount
spec:
  selector:
    matchLabels:
      karpenter.sh/capacity-type: on-demand
  pricePercent: 90
---
# Support for extended resource types (e.g. smarter-devices/fuse)
# https://github.com/kubernetes-sigs/karpenter/issues/751
# https://github.com/kubernetes-sigs/karpenter/issues/729
kind: NodeOverlay
metadata:
  name: discount
spec:
  selector:
    matchLabels:
      karpenter.sh/capacity-type: on-demand
    matchExpressions:
    - key: node.kubernetes.io/instance-type
      operator: In
      values:
      - m5.large
      - m5.2xlarge
      - m5.4xlarge
      - m5.8xlarge
      - m5.12xlarge
  capacity:
    smarter-devices/fuse: 1
---
# Add memory overhead of 10Mi to all instances with 2Gi memory
# https://github.com/aws/karpenter-provider-aws/issues/5161
kind: NodeOverlay
metadata:
  name: discount
spec:
  selector:
    matchLabels:
      karpenter.k8s.aws/instance-memory: 2048
  overhead:
    memory: 10Mi
---

How was this change tested?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@ellistarn ellistarn marked this pull request as draft June 6, 2024 22:18
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Jun 6, 2024
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ellistarn

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 6, 2024
@k8s-ci-robot k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 6, 2024
@k8s-ci-robot k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 6, 2024
@ellistarn ellistarn changed the title feat: Implemented NodeOverlay with support for custom pricing feat: Implemented NodeOverlay with support for custom pricing, resources, and labels Jun 6, 2024
@ellistarn ellistarn changed the title feat: Implemented NodeOverlay with support for custom pricing, resources, and labels feat: Implemented NodeOverlay with support for custom pricing, capacity, overhead, and labels Jun 6, 2024
@ellistarn ellistarn changed the title feat: Implemented NodeOverlay with support for custom pricing, capacity, overhead, and labels feat: Implemented NodeOverlay with support for custom pricing, capacity, overhead, and requirements Jun 6, 2024
@njtran njtran mentioned this pull request Jun 7, 2024
Closed
3 tasks
@sftim
Copy link

sftim commented Jun 10, 2024

(nit) In the PR description, the NodeOverlays all have the same .metadata.name

@sftim
Copy link

sftim commented Jun 10, 2024

Is this an overlay for Nodes, or an overlay for NodePools (maybe more than one NodePool, if we support matching via a selector)?

@jwcesign
Copy link
Contributor

In a real scenario with a saving plan, there is a commitment to a consistent amount of usage. Once the committed usage is exhausted, the price will revert to the on-demand rate. Does this PR take that into consideration?

What I expect is:

  1. When there is remaining committed usage, Karpenter should continue to select on-demand instances(instance type) under the saving plan.
  2. When the committed usage is exhausted, Karpenter should choose spot instances.

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 11, 2024
@jonathan-innis jonathan-innis mentioned this pull request Jun 11, 2024
Closed
@daimaxiaxie
Copy link
Contributor

@ellistarn We redefined the value of topology.kubernetes.io/zone. However, the current zone value is provided by the CloudProvider. Is it possible to support custom topology.kubernetes.io/zone and other WellKnownLabel values?

@sftim
Copy link

sftim commented Jun 12, 2024

@ellistarn We redefined the value of topology.kubernetes.io/zone. However, the current zone value is provided by the CloudProvider. Is it possible to support custom topology.kubernetes.io/zone and other WellKnownLabel values?

It sounds like you don't agree with the zone naming used in an existing integration. You can:

  • use a different label
    • this is the easy and recommended option
  • replace the cloud provider integration code with your own
    (Karpenter, cloud controller manager, etc)

I don't think this PR would need to change to accommodate the ask.

@ellistarn ellistarn mentioned this pull request Jun 12, 2024
Open
@GnatorX
Copy link

GnatorX commented Jun 12, 2024

Thanks for trying to tackle this!

I wonder if there is a cleaner way to represent the overlays to reduce sprawl.

Thinking more generally, each dimension can be additive, adding a new custom resource, and/or adjustment, price changes or resource modification. (Assuming we aren't supporting deleting properties at this time).

Price is a slight outlier from resources so we can separate out. Would it make sense to have at the highest level price and resources. Then subsection of adjustment and additions. Adjustment would support +/- percent or values and additions would just be a dict of new custom resources and values?

@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 12, 2024
@daimaxiaxie
Copy link
Contributor

@sftim I mean nodeoverlay can provide more overlay options, not just resources. For example, Lables.

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 18, 2024
@coveralls
Copy link

Pull Request Test Coverage Report for Build 9882748974

Details

  • 0 of 60 (0.0%) changed or added relevant lines in 1 file are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage decreased (-0.4%) to 77.764%
Changes Missing Coverage Covered Lines Changed/Added Lines %
pkg/apis/v1alpha1/zz_generated.deepcopy.go 0 60 0.0%
Totals Coverage Status
Change from base Build 9880691959: -0.4%
Covered Lines: 8750
Relevant Lines: 11252
💛 - Coveralls

@jmdeal jmdeal mentioned this pull request Jul 15, 2024
Closed
@daverin
Copy link

daverin commented Jul 18, 2024

Hi All

I'm a bit confused about the relationship between kubernetes-sigs/karpenter and aws/karpenter-provider-aws.

The lack of support for custom resource requests/limits is blocking me from autocalling with xilinx devices.

I want to understand how to deploy this branch to EKS.

Any help or understanding would be appreciated.

@njtran njtran mentioned this pull request Jul 30, 2024
Closed
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 2, 2024

This PR has been inactive for 14 days. StaleBot will close this stale PR after 14 more days of inactivity.

@github-actions github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 2, 2024
@sftim
Copy link

sftim commented Aug 2, 2024

I want to understand how to deploy this branch to EKS.

This branch adds a “request for comments“. @ellistarn should the RFC and implementation land as one commit? Seems unusual.

@github-actions github-actions bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 3, 2024
@bnetzi bnetzi mentioned this pull request Aug 7, 2024
Open
@github-actions GitHub Actions
Copy link

This PR has been inactive for 14 days. StaleBot will close this stale PR after 14 more days of inactivity.

@github-actions github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 17, 2024
@jonathan-innis jonathan-innis mentioned this pull request Aug 26, 2024
Open
@github-actions github-actions bot closed this Sep 1, 2024
@zaafar
Copy link

zaafar commented Sep 1, 2024

i think this shouldn't be closed.

@njtran njtran reopened this Sep 4, 2024
@njtran
Copy link
Contributor

njtran commented Sep 4, 2024

/lifecycle frozen

@k8s-ci-robot
Copy link
Contributor

@njtran: The lifecycle/frozen label cannot be applied to Pull Requests.

In response to this:

/lifecycle frozen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@njtran njtran removed lifecycle/closed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Sep 4, 2024
@daimaxiaxie
Copy link
Contributor

What is the progress now? Why does it seems not to be updated.

@github-actions GitHub Actions
Copy link

This PR has been inactive for 14 days. StaleBot will close this stale PR after 14 more days of inactivity.

@github-actions github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 28, 2024
@jmdeal jmdeal mentioned this pull request Oct 1, 2024
Merged
3 tasks
@njtran njtran removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 2, 2024
@zaafar
Copy link

zaafar commented Oct 6, 2024 •
edited
Loading

commenting to make it not stale. this is a very important feature that will take Karpenter to the next level.

@abbas-square
Copy link

Is there any update on this?

@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 8, 2024

This PR has been inactive for 14 days. StaleBot will close this stale PR after 14 more days of inactivity.

@github-actions github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jonathan-innis jonathan-innis jonathan-innis left review comments

@jmdeal jmdeal jmdeal left review comments

@sftim sftim sftim left review comments

@jwcesign jwcesign jwcesign left review comments

@tallaxes tallaxes tallaxes left review comments

@njtran njtran njtran left review comments

@chomatdam chomatdam chomatdam left review comments

@GnatorX GnatorX GnatorX left review comments

@akestner akestner akestner left review comments

@wmgroot wmgroot wmgroot left review comments

@jackfrancis jackfrancis Awaiting requested review from jackfrancis

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.